Development of Social Media Projects with Camel

DevNation / 14th of April - 2014

Charles Moulliard
Architect, Engineer & Committer

Speaker

  • Agricultural Engineer & Zoologist

    • 19 years of experience in IT world development

    • Project manager in Bank, Financial, Telco world

  • Specialized in new technologies Web & Integration

  • Architect/Engineer @Red Hat

  • Committer : Apache ServiceMix, Karaf (PMC), Camel (PMC), Fabric8, Drools, Hawtio, Asciidoctor

180

Speaker - con’t

35

Twitter : http://twitter.com/cmoulliard

35

LinkedIn : http://www.linkedin.com/in/charlesmoulliard

35

Blog : http://cmoulliard.github.io

35

Slideshare : http://www.slideshare.net/cmoulliard

Country where I live

belgium1

atomium

manneken

brussels

brugge

What you should know about Belgium

frits
chocolate
beers
bande dessinee

My Passion(s)

photography

stumpjumper

nature

formula1

Agenda

  • Introduction

  • Social Media Hype

  • Use cases covered

  • Integration & Camel

  • Technology

  • What JBoss Fuse can offer

  • Conclusion

Introduction

500

  • Web 2.0 “Revolution”, User is an actor collaborating

  • Consequences :

    • HTTPs requests 350

    • Volume of data exchanged

Introduction - con’t

150 150 150

  • Technology Transformation

    • HTML5

    • JavaScript / JSon / NoSQL

    • Push    Ajax - WebSocket

350
270
220

Agenda

  • Introduction

  • Social Media Hype

  • Use cases covered

  • Integration & Camel

  • Technology

  • What JBoss Fuse can offer

  • Conclusion

Social Media Hype

Social-network ilike Social-Media-Marketing

  • Business - Companies

    • Marketing Strategy,

    • Product Promotion,

    • Launch or Campaign annoucement

    • Seeking candidates

  • Use Web2.0 but Social Medias

Social Media Hype - con’t

Why    To create content that attracts attention & encourages readers to share it with their social networks

Social Media
social solutions

Agenda

  • Introduction

  • Social Media Hype

  • Use cases covered

  • Integration & Camel

  • Technology

  • What JBoss Fuse can offer

  • Conclusion

Use cases covered - demo1

  • Pattern : Real Time & Broadcasting

  • Definition : Collect data in real time from different providers (twitter, facebook, …) and broadcast/multicast data to subscribed channels

  • Business : Event, conference, meeting room

550

Use cases covered - demo2

  • Pattern : Collect Data Metrics/Statistics

  • Definition : Retrieve data from Social Media and store them in order to query the data to analyze the results

  • Business : Measure performances of a campaign, product launch, analyze data to design marketing strategies

300

Use cases covered - demo3

  • Pattern : Business Activity Monitoring

  • Definition : Monitor activities, operational days using reporting, dashboard tools.

  • Business : Measure performances, rentability, return of investment

analyze blog content

Agenda

  • Introduction

  • Social Media Hype

  • Use cases covered

  • Integration & Camel

  • Technology

  • What JBoss Fuse can offer

  • Conclusion

Integration

hub

  • Integration is    really hard

  •    protocols, standards, data formats, systems

  • Long story since Common Object Request Broker Architecture

Integration - con’t

producer-consumer

  • Concept

    • Decouple Producer from Consumer

    • Message transport information

Integration - con’t

bus

  •    Layer to transport messages : BUS

  •    ESB, JBI, SCA

jbi

Integration - con’t

camel box small
  • OpenSource Java Integration Framework

  • Designed around : Domain Specific Language

  • Implement Enterprise Integration Patterns

book

Camel

  • > 50 patterns implemented

  • and more : Loadbalancer, Throttler, Delayer, …

patterns
patterns 3

Camel - con’t

factory

  • Key features

    • Component

    • Endpoint

    • Consumer

    • Producer

Camel - con’t

  • Key features : route, processor

pipeline

Camel - con’t

  • Interceptor : trace, log, capture business events

pipeline2

Camel - con’t

camel-features

  • Container for the routes    CamelContext

Camel - con’t

camel-features2

  • Cross communication not allowed using direct, seda

Camel - con’t

camel-features3

  • But possible Using BUS like NMR, Broker, Shared Component with direct-vm, vm

Camel - con’t

type-converter

  • Type Converter Strategy

  • Allow to convert the body payloads from one type to another

  • To and From these types

    • File

    • String

    • byte[] and ByteBuffer

    • InputStream and OutputStream

    • Reader and Writer

Camel - con’t

  • Data Transformation for complex use case

package org.devnation.camel;

import java.io.InputStream;
import java.io.OutputStream;
import org.apache.camel.Exchange;

public interface DataFormat {

    void marshal(Exchange exchange, Object graph, OutputStream stream) throws Exception;

    Object unmarshal(Exchange exchange, InputStream stream) throws Exception;
}
  • Marshalling : Object XML (JAXB)

  • Unmarshalling : XML Object (JAXB)

Camel - con’t

dataformat 2

Camel - con’t

inonly

Camel - con’t

inout

Camel - con’t

  • Fluent API

package org.devnation.camel;

import org.apache.camel.builder.RouteBuilder;

public class ExampleRouteBuilder extends RouteBuilder {
    @Override
    public void configure() throws Exception {

        from("amq:queue:quotes")
           .filter().xpath("/quote/product/ = 'widget")
                .bean("QuotesService", "widget")
           .filter().xpath("/quote/product/ = 'gadget")
                .bean("QuotesService","gadget");

    }
}

Camel - con’t

  • Alternative : Spring, Blueprint DSL

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="
       http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
       http://camel.apache.org/schema/spring http://camel.apache.org/schema/spring/camel-spring.xsd
    ">
    <bean id="quotesService" class="org.devnation.camel.QuotesService"/>"

    <camelContext  xmlns="http://camel.apache.org/schema/spring">
        <route>
            <from uri="amq:queue:quotes"/>
            <filter>
                <xpath>"/quote/product/ = 'widget"</xpath>
            </filter>
                <bean id="quotesService" method="widget"/>
            <filter>
                <xpath>"/quote/product/ = 'gadget"</xpath>
            </filter>
            <bean id="quotesService" method="gadget"/>
        </route>
    </camelContext>

</beans>

Camel - con’t

  • In memory bus / alternative to JBI, SCA, NMR

  • Transport objects : XML, File, Stream, Bytes

  • Predicate & Expression language (xslt, xpath, …)

  • Sync/Async exchanges

  • Threads Management,

  • Tx Architecture

  • Error and exception handling

  • Policy driven

  • Container agnostic

Camel - con’t

components

Agenda

  • Introduction

  • Social Media Hype

  • Use cases covered

  • Integration & Camel

  • Technology

  • What JBoss Fuse can offer

  • Conclusion

Technology - Camel

social components

combined with

  • camel-gmail

  • camel-dropbox

  • camel-ajax, camel-websocket,

  • camel-sql, camel-jdbc, …

  • camel-elasticsearch

Technology - Common

Open standard for Authorization

OAuth 75

oauth

Technology - Camel

        // start with a 30 day window for the first delayed poll
        String since = "RAW("
                + new SimpleDateFormat(FACEBOOK_DATE_FORMAT).format(
                new Date(System.currentTimeMillis() - TimeUnit.MILLISECONDS.convert(30, TimeUnit.DAYS)))
                + ")";

        from("facebook://searchPosts?reading.limit=10&" +
                "reading.locale=en.US&reading.since=" + since + "&" +
                "consumer.initialDelay=1000&" +
                "consumer.sendEmptyMessageWhenIdle=true&"
                + getOauthParams());

Technology - Camel

  • OAuth

    protected String getOauthParams() {
        return "oAuthAppId=" + properties.get("oAuthAppId") +
                "&oAuthAppSecret=" + properties.get("oAuthAppSecret")
                + (properties.get("oAuthAccessToken") != null
                ? ("&oAuthAccessToken=" + properties.get("oAuthAccessToken")) : "");
    }
  • Externalize tokens parameters with Property PlaceHolder

consumer.key=rtPyyyyyyyyyyyyyyyyg
consumer.secret=HyoWW3xxxxxxxxxxxxxxxxTuyEz8qrk
access.token=7976eeeeeeeeeeeeeeeeeeeuRmG0IP
access.token.secret=VNtttttttttttttttttg0B8jWGs

Technology - Camel

Example :

  • To create a post within your Facebook profile

  • Send a facebook4j.PostUpdate body to a camel facebook producer

        PostUpdate post = new PostUpdate(new URL("http://facebook4j.org"))
                .picture(new URL("http://facebook4j.org/images/hero.png"))
                .name("Facebook4J - A Java library for " +
                        "the Facebook Graph API")
                .caption("facebook4j.org")
                .description("Facebook4J is a Java library" +
                        " for the Facebook Graph API.");

        from("direct:post")
                .setBody().constant(post)
                .to("facebook://postFeed/inBody=postUpdate");

Technology - Camel

Example

        from("twitter://search?type=polling&delay="
                + delay + "&useSSL=true&keywords="
                + keywords + "&" + getUriTokens())
                .choice()
                   .when().simple("${body.tweet.text} > 'java'")
                      .bean("Service", "push")
                .otherwise()
                      .to(">> Tweets received");

Technology - Camel

OAuth

    protected String getUriTokens() {
        return "consumerKey=" + consumerKey
                + "&consumerSecret=" + consumerSecret
                + "&accessToken=" + accessToken
                + "&accessTokenSecret=" + accessTokenSecret;
    }

Advanced Query

Technology - Camel

Simple Example to post a tweet

        Date now = new Date();
        String tweet = "Demo: this is a tweet posted on " + now.toString();

        from("direct:post")
            .setBody().constant(tweet)
            .to("twitter://timeline/user?" + getUriTokens());

Technology - Camel

  • Object / JSON

  • Use utility like Google JSON API

  • Marshall/Unmmarshall JSON

Twitter - con’t

BAD -   

String data = "{ " +
    " \"" + "timestamp" + "\" : \"" + formatDate(generateTimeStamp()) + "\"," +
    " \"" + "createdAt" + "\" : \"" + status.getCreatedAt().toString() + "\"," +
    " \"" + "id" + "\" : \"" + status.getId() + "\"," +
    " \"" + "text" + "\" : \"" + status.getText() + '\'' + "\"," +
    " \"" + "isFavorited" + "\" : \"" + status.isFavorited() + "\"," +
    " \"" + "isRetweeted" + "\" : \"" + status.isRetweeted() + "\"," +
    " \"" + "favoriteCount" + "\" : \"" + status.getFavoriteCount() + "\"," +
    " \"" + "inReplyToScreenName" + "\" : \"" + status.getInReplyToScreenName() + '\'' + "\"," +
    " \"" + "geoLocation" + "\" : \"" + status.getGeoLocation() + "\"," +
    " \"" + "place" + "\" : \"" + status.getPlace() + "\"," +
    " \"" + "retweetCount" + "\" : \"" + status.getRetweetCount() + "\"," +
    " \"" + "isoLanguageCode" + "\" : \"" + status.getIsoLanguageCode() + "\"," +
    " \"" + "user" + "\" : \"" + status.getUser().getName() + "\"," +
    " \"" + "country" + "\" : \"" + country + "\"" +
    " }";

Technology - con’t

BETTER -   

  private void init(JSONObject json) throws TwitterException {
  	id = getLong("id", json);
  	source = getUnescapedString("source", json);
  	createdAt = getDate("created_at", json);
  	isTruncated = getBoolean("truncated", json);
  	inReplyToStatusId = getLong("in_reply_to_status_id", json);
  	inReplyToUserId = getLong("in_reply_to_user_id", json);
  	isFavorited = getBoolean("favorited", json);

Technology - Camel

Routing engine Camel & EIP Patterns like   

Transform

150

Multicast

150

Technology - Real Time

websocket

  • Websocket

    • Full-duplex single
      socket connection

    • HTTP request followed
      by WebSocket data
      Packets exchange

    • ws:// and wss:// protocol

    • Part of HTML5 initiative

    • Specification rfc-6455 (Dec-2011) managed by IETF

Technology - Real Time - DEMO1

  • Integrated within ActiveMQ & Camel

550

Technology - Data Metrics

elasticsearch1 lucene_logo nosql

  • Kibana & ElasticSearch    Insight Technology

  • Collect Logs, Camel metrics …

Technology - Data Metrics

400

Use cases covered - Data Metrics

elasticsearch

Technology - Data Metrics

  • Create a bean recuperating Message/Exchange using @Header, @Body

  • Store it using org.fusesource.insight.storage.StorageService

public class StoreService {

    private static String ES_TYPE = "insight-tweet";
    private static StorageService storageService;

    public static void store(@Header("tweet-full") String data) {
        storageService.store(ES_TYPE, generateTimeStamp(), data);
    }

    public void setStorageService(StorageService storageService) {
        StoreService.storageService = storageService;
    }

Technology - Data Metrics

  • Inject Storage service

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="
           http://www.osgi.org/xmlns/blueprint/v1.0.0
             http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd">

    <!-- Service Service used to send tweets to ES Storage, parse JSON -->
    <bean id="helper" class="org.devnation.demo.camel.Service">
        <property name="storageService">
            <reference interface="org.fusesource.insight.storage.StorageService" />
        </property>
    </bean>

</blueprint>

Technology - Data Metrics

  • Call it from your Camel route

  from("twitter://search?type=polling&delay=" + delay
          + "&useSSL=true&keywords=" + keywords
          + "&" + getUriTokens())
          .routeId("Tweet-Store-WS")
          .delay(5000)

          // We receive the Twitter4J Status that we will use
          // to extract info text, isfavorited, is Retweeted, geolocation, isoLanguageCode, contributors
          // and create A JSON message used later on to store it in ES
          .setHeader("tweet-full").method(Service.class, "getJSONTweet")

          // Message is stored using insight in ElasticSearch
          .bean(Service.class, "store")

Technology - Data Metrics - DEMO2

eshead

Technology - Dashboard

80 Hawtio

  • Lightweight & modular HTML5 web console with plugins for managing Java MBeans

  • Javascript / REST front end    jolokia JMX translator

100    100    150

  • Heart of the new Fuse Management Console

Technology - Dashboard

  • Extend Hawtio Dashboard

dashboard trends

Technology - Dashboard

  • Declare MBean Interface

public interface SocialMediaMBean {

    /* Attributes */
    void setTweetsCounter(Integer val);
    public Integer getTweetsCounter();

    /* Operations */
    List<String> searchTweets(String keywords) throws TwitterException;
    String userInfo(String id) throws TwitterException;

Technology - Dashboard

  • Service Implementation & MBean registration

    public List<String> searchTweets(String keywords) throws TwitterException {
        return twitterService.searchTweets(keywords);
    }

    public void init() {
        try {
            if (objectName == null) {
                objectName = new ObjectName("hawtio:type=SocialMedia");
            }
            if (mBeanServer == null) {
                mBeanServer = ManagementFactory.getPlatformMBeanServer();
            }
            try {
                mBeanServer.registerMBean(this, objectName);
            } catch (InstanceAlreadyExistsException iaee) {
                // Try to remove and re-register
                LOG.info("Re-registering SchemaLookup MBean");
                mBeanServer.unregisterMBean(objectName);
                mBeanServer.registerMBean(this, objectName);
            }
        } catch (Exception e) {
            LOG.warn("Exception during initialization: ", e);
            throw new RuntimeException(e);
        }
    }

Technology - Dashboard - DEMO3

  • Inject Service & init beans

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0">

    <cm:property-placeholder id="twitterConfig" persistent-id="twitter"
               update-strategy="reload"/>

    <bean id="twitterFactory" class="org.devnation.demo.service.TwitterFactory">
        <property name="consumerKey" value="${consumer.key}"/>
        <property name="consumerSecret" value="${consumer.secret}"/>
        <property name="accessToken" value="${access.token}"/>
        <property name="accessTokenSecret" value="${access.token.Secret}"/>
    </bean>

    <bean id="twitterService" class="org.devnation.demo.service.TwitterService">
        <property name="twitterFactory" ref="twitterFactory"/>
    </bean>

    <bean id="socialDataMBean" class="org.devnation.demo.service.SocialMedia"
          init-method="init" destroy-method="destroy" scope="singleton">
        <property name="twitterService" ref="twitterService"/>
    </bean>
</blueprint>

Technology - Plugin

  • Create your own hawtio plugin

  • Web project (war, bundle)

  • Require to register io.hawt.web.plugin.HawtioPlugin with parameters

    • Context

    • JS Scripts location

    • Name and domain (used by controller)

Technology - Plugin

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0">

    <bean id="plugin" class="io.hawt.web.plugin.HawtioPlugin" init-method="init" destroy-method="destroy">
        <property name="name" value="${plugin-name}"/>
        <property name="context" value="${plugin-context}"/>
        <property name="domain" value="${plugin-domain}"/>
        <property name="scripts" value="${plugin-scripts}"/>
    </bean>

</blueprint>

Technology - Plugin

  • Design Front end (AngularJS)

  • Send REST request to call Service

80

        $scope.searchTweets = function () {
            if (Core.isBlank($scope.keywords)) {
                return;
            }
            jolokia.request({
                type: 'exec',
                mbean: SOCIAL.mbean,
                operation: 'searchTweets',
                arguments: [$scope.keywords]
            }, {
                method: 'POST',
                success: function (response) {
                    /* Simple Table */
                    $scope.tweets = response.value.map(function (val) {
                        return { tweet: val };
                    });
                    Core.$apply($scope);
                },
                error: function (response) {
                    SOCIAL.log.warn("Failed to search for Tweets: ", response.error);
                    SOCIAL.log.info("Stack trace: ", response.stacktrace);
                    Core.$apply($scope);
                }

Technology - Plugin

<div class="row-fluid gridStyle" ng-controller="SOCIAL.FormController">
    <h3>Twitter</h3>

    <div class="span12">
        <div>
            <form name="inputForm" ng-submit="searchTweets()">
                <div>
                    Keyword(s) : <input type="text" class="entry-widget" ng-model="keywords">
                </div>
                <p class="span4 centered">
                    <input type="submit" class="btn btn-success" ng-disabled="!keywords" value="Search information">
                </p>
            </form>
        </div>
    </div>

    <div class="span8">
        <div class="gridStyle" ng-grid="tweetsGrid"/>
    </div>

</div>

Technology - Plugin - DEMO3

plugin query

Agenda

  • Introduction

  • Social Media Hype

  • Use cases covered

  • Integration & Camel

  • Technology

  • What JBoss Fuse can offer

  • Conclusion

What JBoss Fuse can offer

basics 1
150

What JBoss Fuse can offer - con’t

basics 2
150

What JBoss Fuse can offer - con’t

advances 1
150

Technology - con’t

advances 2
150

Agenda

  • Introduction

  • Social Media Hype

  • Integration & Camel

  • Use cases covered

  • Technology

  • What JBoss Fuse can offer

  • Conclusion

Conclusion

  • Fuse Technology ready    design Social Media Projects

  • To cover use cases   

    • Real Time,

    • Data collection,

    • Fuse Analytics,

    • BAM

Questions

questions